Class Activation Mapping Research Articles

Efficient Task-driven Video Data Privacy Protection for Smart Camera Surveillance System

As one of the most commonly used AIoT sensors, smart cameras and their supporting services, namely cloud video surveillance (CVS) systems, have brought great convenience to people’s lives. Recent CVS providers use different machine learning techniques to improve their services (regarded as tasks) based on the uploaded video. However, uploading data to the CVS providers may cause severe privacy issues. Existing works that remove privacy information could not achieve a high tradeoff between data usability and privacy, because the importance of information varies with the task. In addition, it is challenging to design a real-time privacy protection mechanism, especially in resource-constrained smart cameras. In this work, we design a task-driven and efficient video privacy protection mechanism for a better tradeoff between privacy and data usability. We use Class Activation Mapping to protect privacy while preserving data usability. To improve the efficiency, we utilize the motion vector and residual matrix produced during video codec. Our work outperforms the region of interest–based methods in data protection while preserving data usability. The attack accuracy drops 70%, while the task accuracy is comparable to those without protection (within ± 4%). The average protection frame rate of the High Definition video can exceed 16 fps+ even on a CPU.

Enhancing brain tumor detection in MRI images through explainable AI using Grad-CAM with Resnet 50

This study addresses the critical challenge of detecting brain tumors using MRI images, a pivotal task in medical diagnostics that demands high accuracy and interpretability. While deep learning has shown remarkable success in medical image analysis, there remains a substantial need for models that are not only accurate but also interpretable to healthcare professionals. The existing methodologies, predominantly deep learning-based, often act as black boxes, providing little insight into their decision-making process. This research introduces an integrated approach using ResNet50, a deep learning model, combined with Gradient-weighted Class Activation Mapping (Grad-CAM) to offer a transparent and explainable framework for brain tumor detection. We employed a dataset of MRI images, enhanced through data augmentation, to train and validate our model. The results demonstrate a significant improvement in model performance, with a testing accuracy of 98.52% and precision-recall metrics exceeding 98%, showcasing the model’s effectiveness in distinguishing tumor presence. The application of Grad-CAM provides insightful visual explanations, illustrating the model’s focus areas in making predictions. This fusion of high accuracy and explainability holds profound implications for medical diagnostics, offering a pathway towards more reliable and interpretable brain tumor detection tools.

Keratoconus Progression Determined at the First Visit: A Deep Learning Approach With Fusion of Imaging and Numerical Clinical Data.

Multiple clinical visits are necessary to determine progression of keratoconus before offering corneal cross-linking. The purpose of this study was to develop a neural network that can potentially predict progression during the initial visit using tomography images and other clinical risk factors. The neural network's development depended on data from 570 keratoconus eyes. During the initial visit, numerical risk factors and posterior elevation maps from Scheimpflug imaging were collected. Increase of steepest keratometry of 1 diopter during follow-up was used as the progression criterion. The data were partitioned into training, validation, and test sets. The first two were used for training, and the latter for performance statistics. The impact of individual risk factors and images was assessed using ablation studies and class activation maps. The most accurate prediction of progression during the initial visit was obtained by using a combination of MobileNet and a multilayer perceptron with an accuracy of 0.83. Using numerical risk factors alone resulted in an accuracy of 0.82. The use of only images had an accuracy of 0.77. The most influential risk factors in the ablation study were age and posterior elevation. The greatest activation in the class activation maps was seen at the highest posterior elevation where there was significant deviation from the best fit sphere. The neural network has exhibited good performance in predicting potential future progression during the initial visit. The developed neural network could be of clinical significance for keratoconus patients by identifying individuals at risk of progression.

Exploring the influence of attention for whole-image mammogram classification

Attention is an important component of modern Convolutional Neural Networks (CNNs) that has been shown to improve baseline model performance for a wide variety of tasks. Attention has shown specific promise in the classification and segmentation of mammograms, but we have a limited understanding of why attention improves performance in these domains. In this paper, we present a robust comparison of different combinations of baseline models and attention methods at two resolutions for whole mammogram classification of masses and calcifications. We find that attention generally helps to improve baseline model performance. However, the extent of improvement is governed by a combination of model architecture and the statistical characteristics of the data. Specifically, we show that high amounts of pooling and model complexity may result in decreased performance for data with high variability. To better understand the effect of attention on mammogram classification, we used LayerCAM, a hierarchical Class Activation Map (CAM) approach, to visualize where the network pays attention in the input image. This research provides statistical evidence that attention can improve the correlation between model performance and LayerCAM activation in the region of interest (ROI). However, these correlations are weak and variable, indicating that improvements in model performance due to attention are not necessarily caused by increased model activation near the ROI. Overall, our work provides novel insights to help guide future efforts in incorporating attention-based mechanisms for mammogram classification.

Robust deep learning method for fruit decay detection and plant identification: enhancing food security and quality control

This paper presents a robust deep learning method for fruit decay detection and plant identification. By addressing the limitations of previous studies that primarily focused on model accuracy, our approach aims to provide a more comprehensive solution that considers the challenges of robustness and limited data scenarios. The proposed method achieves exceptional accuracy of 99.93%, surpassing established models. In addition to its exceptional accuracy, the proposed method highlights the significance of robustness and adaptability in limited data scenarios. The proposed model exhibits strong performance even under the challenging conditions, such as intense lighting variations and partial image obstructions. Extensive evaluations demonstrate its robust performance, generalization ability, and minimal misclassifications. The inclusion of Class Activation Maps enhances the model’s capability to identify distinguishing features between fresh and rotten fruits. This research has significant implications for fruit quality control, economic loss reduction, and applications in agriculture, transportation, and scientific research. The proposed method serves as a valuable resource for fruit and plant-related industries. It offers precise adaptation to specific data, customization of the network architecture, and effective training even with limited data. Overall, this research contributes to fruit quality control, economic loss reduction, and waste minimization.

A deep learning approach for Maize Lethal Necrosis and Maize Streak Virus disease detection

Maize is an important crop cultivated in Sub-Saharan Africa, essential for food security. However, its cultivation faces significant challenges due to debilitating diseases such as Maize Lethal Necrosis (MLN) and Maize Streak Virus (MSV), which can lead to severe yield losses. Traditional plant disease diagnosis methods are often time-consuming and prone to errors, necessitating more efficient approaches. This study explores the application of deep learning, specifically Convolutional Neural Networks (CNNs), in the automatic detection and classification of maize diseases. We investigate six architectures: Basic CNN, EfficientNet V2 B0 and B1, LeNet-5, VGG-16, and ResNet50, using a dataset of 15344 images comprising MSV, MLN, and healthy maize leaves. Additionally, We performed hyperparameter tuning to improve the performance of the models and Gradient-weighted Class Activation Mapping (Grad-CAM) for model interpretability. Our results show that the EfficientNet V2 B0 model demonstrated an accuracy of 99.99% in distinguishing between healthy and disease-infected plants. The results of this study contribute to the advancement of AI applications in agriculture, particularly in diagnosing maize diseases within Sub-Saharan Africa.

Methodology for Evaluating the Generalization of ResNet

Convolutional neural networks (CNNs) have achieved promising results in many tasks, and evaluating the model’s generalization ability based on the trained model and training data is paramount for practical applications. Although many measures for evaluating the generalization of CNN models have been proposed, the existing works are limited to small-scale or simplified model sets, which would result in poor accuracy and applicability of the derived methods. This study addresses these limitations by leveraging ResNet models as a case study to evaluate the model’s generalization ability. We utilized Intersection over Union (IoU) as a method to quantify the ratio of task-relevant features to assess model generalization. Class activation maps (CAMs) were used as a representation of the distribution of features learned by the model. To systematically investigate the generalization ability, we constructed a diverse model set based on the ResNet architecture. A total of 2000 CNN models were trained on the ImageNet subset by systematically changing commonly used hyperparameters. The results of our experiments revealed a strong correlation between the IoU-based evaluation method and the model’s generalization performance (Pearson correlation coefficient more than 0.8). We also performed extensive experiments to demonstrate the feasibility and robustness of the evaluation methods.

Toward explainable AI in radiology: Ensemble-CAM for effective thoracic disease localization in chest X-ray images using weak supervised learning

Chest X-ray (CXR) imaging is widely employed by radiologists to diagnose thoracic diseases. Recently, many deep learning techniques have been proposed as computer-aided diagnostic (CAD) tools to assist radiologists in minimizing the risk of incorrect diagnosis. From an application perspective, these models have exhibited two major challenges: (1) They require large volumes of annotated data at the training stage and (2) They lack explainable factors to justify their outcomes at the prediction stage. In the present study, we developed a class activation mapping (CAM)-based ensemble model, called Ensemble-CAM, to address both of these challenges via weakly supervised learning by employing explainable AI (XAI) functions. Ensemble-CAM utilizes class labels to predict the location of disease in association with interpretable features. The proposed work leverages ensemble and transfer learning with class activation functions to achieve three objectives: (1) minimizing the dependency on strongly annotated data when locating thoracic diseases, (2) enhancing confidence in predicted outcomes by visualizing their interpretable features, and (3) optimizing cumulative performance via fusion functions. Ensemble-CAM was trained on three CXR image datasets and evaluated through qualitative and quantitative measures via heatmaps and Jaccard indices. The results reflect the enhanced performance and reliability in comparison to existing standalone and ensembled models.

Weakly supervised semantic segmentation based on superpixel affinity

Weakly supervised semantic segmentation (WSSS) usually employs the method of modifying and extending class activation map (CAM) seeds to achieve semantic segmentation. However, the down-sampling of the network weakens the edge awareness of CAMs, leading to under- or over-activation problems, which affects the segmentation quality. Considering the excellent contour attachment property of superpixels and the high semantic similarity between pixels within the same superpixel, we propose a superpixel affinity-based method that uses multi-scale features to aggregate superpixels with the same semantics, providing complete localization supervision for the generation of CAMs. In order to improve the accuracy of semantic labels for superpixels, we utilize a method of deep feature reorganization to improve the quality of network-generated CAM seeds. The experimental results indicate that the proposed method has achieved satisfactory performance on PASCAL VOC 2012 and MS COCO 2014 datasets.

Automatic classification of spinal osteosarcoma and giant cell tumor of bone using optimized DenseNet

ObjectiveThis study aims to explore an optimized deep-learning model for automatically classifying spinal osteosarcoma and giant cell tumors. In particular, it aims to provide a reliable method for distinguishing between these challenging diagnoses in medical imaging. MethodsThis research employs an optimized DenseNet model with a self-attention mechanism to enhance feature extraction capabilities and reduce misclassification in differentiating spinal osteosarcoma and giant cell tumors. The model utilizes multi-scale feature map extraction for improved classification accuracy. The paper delves into the practical use of Gradient-weighted Class Activation Mapping (Grad-CAM) for enhancing medical image classification, specifically focusing on its application in diagnosing spinal osteosarcoma and giant cell tumors. The results demonstrate that the implementation of Grad-CAM visualization techniques has improved the performance of the deep learning model, resulting in an overall accuracy of 85.61%. Visualizations of images for these medical conditions using Grad-CAM, with corresponding class activation maps that indicate the tumor regions where the model focuses during predictions. ResultsThe model achieves an overall accuracy of 80% or higher, with sensitivity exceeding 80% and specificity surpassing 80%. The average area under the curve AUC for spinal osteosarcoma and giant cell tumors is 0.814 and 0.882, respectively. The model significantly supports orthopedics physicians in developing treatment and care plans. ConclusionThe DenseNet-based automatic classification model accurately distinguishes spinal osteosarcoma from giant cell tumors. This study contributes to medical image analysis, providing a valuable tool for clinicians in accurate diagnostic classification. Future efforts will focus on expanding the dataset and refining the algorithm to enhance the model's applicability in diverse clinical settings.

Weakly Supervised Instance Segmentation via Class Double-Activation Maps and Boundary Localization

Weakly supervised instance segmentation based on image-level class labels has recently gained much attention, in which the primary key step is to generate the pseudo labels based on class activation maps (CAMs). Most methods adopt binary cross-entropy (BCE) loss to train the classification model. However, since BCE loss is not class mutually exclusive, activations among classes occur independently. Thus, not only do foreground classes are wrongly activated as background, but also incorrect activations among confusing classes are occurred in the foreground. To solve this problem, we propose the Class Double-Activation Map, called Double-CAM. Firstly, the vanilla CAM is extracted from the multi-label classifier and then fused with the output feature map of backbone. The enhanced feature map of each class is fed into the single-label classification branch with softmax cross-entropy (SCE) loss and entropy minimization module, from which the more accurate Double-CAM is extracted. It refines the vanilla CAM to improve the quality of pseudo labels. Secondly, to mine object edge cues from Double-CAM, we propose the Boundary Localization (BL) module to synthesize boundary annotations, so as to provide constraints for label propagation more explicitly without adding additional supervision. The quality of pseudo masks is also improved substantially with the addition of BL module. Finally, the generated pseudo labels are used to train fully supervised instance segmentation networks. The evaluations on VOC and COCO datasets show that our method achieves excellent performance, outperforming mainstream weakly supervised segmentation methods at the same supervisory level, even those that depend on stronger supervision.

Automated identification of aquatic insects: A case study using deep learning and computer vision techniques

Despite huge attention in other domains, deep learning is only very slowly beginning to be applied in biodiversity research. Mayflies (Ephemeroptera), stoneflies (Plecoptera) and caddisflies (Trichoptera), often abbreviated as EPT, are frequently used for freshwater biomonitoring due to their large numbers and sensitivity to environmental changes. However, the unambiguous morphological identification of EPT species is a challenging, but fundamental task. Morphological identification of these freshwater insects is therefore not only extremely time-consuming and costly, but also often leads to misjudgments or generates datasets with low taxonomic resolution. Here, we investigated the application of deep learning to increase the efficiency and taxonomic resolution of biomonitoring programs. Our database contains 90 EPT taxa (genus or species level), with the number of images per category ranging from 21 to 300 (16,650 in total). Upon completion of training, a CNN (Convolutional Neural Network) model was created, capable of automatically classifying these taxa into their appropriate taxonomic categories with an accuracy of 98.7 %. For the extensive set of 68 tested taxa, our model achieved a perfect classification rate of 100 %. We achieved noteworthy classification accuracy with morphologically closely related taxa within the training data (e.g., species of the genus Baetis, Hydropsyche, Perla). Gradient-weighted Class Activation Mapping (Grad-CAM) visualized the morphological features responsible for the classification of the treated species in the CNN models. In the Ephemeroptera, the head was the most important feature, while the thorax and abdomen were equally important for the classification of Plecoptera taxa. For the order Trichoptera, the head and thorax were almost equally important. Our database is recognized as the most extensive aquatic insect database, notably distinguished by its wealth of included categories (taxa). Our approach can help solve long-standing challenges in biodiversity research and address pressing issues in monitoring programs by saving time in sample and data processing.

Retinal disease diagnosis with unsupervised Grad-CAM guided contrastive learning

To alleviate the reliance on expensive annotations, contrastive learning techniques have been applied to diagnose diseases from various types of medical images. However, popular contrastive learning methods, which generate positive pairs by random cropping, face challenges in diagnosing retinal diseases, due to the retinal disease’s characteristic that retinal lesions are tiny and randomly distributed in abnormal fundus images. These lesions may be missed after random cropping, resulting in semantically inconsistent positive pairs that hinder the effectiveness of contrastive learning for retinal disease diagnosis. To address this issue, we propose a novel unsupervised gradient-weighted class activation mapping (Grad-CAM) strategy to roughly locate lesions, thereby suppressing or even eliminating semantically inconsistent positive pairs. Specifically, we develop a gradient-weighted Class Activation Map guided Contrastive Learning (CAMCL) method with two branches for the Grad-CAM based instance discrimination task and the k-nearest neighbors (KNN)-based cluster-wise discrimination task, respectively. By minimizing the KNN loss, the cluster-wise discrimination branch learns high-level representations containing class semantic information. This is then followed by gradient back-propagation to generate Grad-CAM heatmaps from unlabeled data. The generated heatmaps can highlight class-discriminative regions in abnormal fundus images (e.g., retinal lesions) to identify semantically consistent positive pairs while suppressing inconsistent ones. The semantically consistent positive pairs are then input to the instance discrimination task for contrastive learning. In this manner, the semantic inconsistency problem is relieved, and the improved contrastive learning pipeline can be effectively used for retinal disease diagnosis. Experimental results on five retinal disease classification datasets show that our model surpasses other contrastive learning methods, indicating a promising approach for clinical application.

A Novel Approach of Brain-Computer Interfacing (BCI) and Grad-CAM Based Explainable Artificial Intelligence: Use Case Scenario for Smart Healthcare

BackgroundIn order to push the frontiers of brain-computer interface (BCI) and neuron-electronics, this research presents a novel framework that combines cutting-edge technologies for improved brain-related diagnostics in smart healthcare. This research offers a ground-breaking application of transparent strategies to BCI, promoting openness and confidence in brain-computer interactions and taking inspiration from Grad-CAM (Gradient-weighted Class Activation Mapping) based Explainable Artificial Intelligence (XAI) methodology. The landscape of healthcare diagnostics is about to be redefined by the integration of various technologies, especially when it comes to illnesses related to the brain. New MethodA novel approach has been proposed in this study comprising of Xception architecture which is trained on imagenet database following transfer learning process for extraction of significant features from magnetic resonance imaging dataset acquired from publicly available distinct sources as an input and linear support vector machine has been used for distinguishing distinct classes.Afterwards, gradient-weighted class activation mapping has been deployed as the foundation for explainable artificial intelligence (XAI) for generating informative heatmaps, representing spatial localization of features which were focused to achieve model’s predictions. ResultsThus, the proposed model not only provides accurate outcomes but also provides transparency for the predictions generated by the Xception network to diagnose presence of abnormal tissues and avoids overfitting issues. Hyperparameters along with performance-metrics are also obtained while validating the proposed network on unseen brain MRI scans to ensure effectiveness of the proposed network. Comparison with existing methods and ConclusionsThe integration of Grad-CAM based explainable artificial intelligence with deep neural network namely Xception offers a significant impact in diagnosing brain tumor disease while highlighting the specific regions of input brain MRI images responsible for making predictions. In this study, the proposed network results in 98.92% accuracy, 98.15% precision, 99.09% sensitivity, 98.18% specificity and 98.91% dice-coefficient while identifying presence of abnormal tissues in the brain. Thus, Xception model trained on distinct dataset following transfer learning process offers remarkable diagnostic accuracy and linear support vector act as a classifier to provide efficient classification among distinct classes. In addition, the deployed explainable artificial intelligence approach helps in revealing the reasoning behind predictions made by deep neural network having black-box nature and provides a clear perspective to assist medical experts in achieving trustworthiness and transparency while diagnosing brain tumor disease in the smart healthcare

Fusion of satellite and street view data for urban traffic accident hotspot identification

As the number of vehicles and the volume of traffic swell in urban centers, cities have experienced a concomitant increase in traffic accidents. Proactively identifying accident-prone hotspots in urban environments holds the promise of preventing traffic mishaps, thereby curtailing the incidence of accidents and reducing property damage. This research introduces the Two-Branch Contextual Feature-Guided Converged Network (TCFGC-Net) utilizing multimodal satellite and street view data. Designed to extract global structural features from satellite imagery and dynamic continuous features from street view imagery, the model aims to improve the accuracy of detecting urban accident hotspots. For the satellite imagery branch, we propose the Contextual Feature Coupled Convolutional Neural Network (Trans-CFCCNN) designed to extract global spatial features and discern feature correlations across adjacent regions. For the street view imagery branch, we develop the Sequential Feature Recurrent Attention Network (SFRAN) to assimilate and integrate dynamic scene features captured from successive street view images. We designed the Multi-Branch Feature Adaptive Fusion Structure (MBFAF) to aggregate different branch features for accurate identification of accident hotspots. Experimental results show that the model performs well, with an overall accuracy of 93.7 %. Ablation studies confirm that relative to standalone street view and satellite branch analyses, implementing multimodal fusion enhances the model's accuracy by 12.05 % and 17.86 %, respectively. The innovative fusion structure proposed herein garners a 4.22 % increase in model accuracy, outpacing conventional feature concatenation techniques. Furthermore, the model outperforms existing deep learning models in terms of overall efficacy. Additionally, to showcase the efficacy of the proposed model structure, we utilize Class Activation Maps (CAM) to provide visual interpretability for the model. These results suggest that the dual-branch fusion model effectively decreases false alarm occurrences and directs the model's focus toward regions more pertinent to accident hotspots. Finally, the code and model used for identifying hotspots of urban traffic accidents in this study are available for access: https://github.com/gwt-ZJU/TCFGC-Net.

A discrepancy-aware self-distillation method for multi-modal glioma grading

Gliomas are the most common intracranial primary tumors. Accurate grading of gliomas is crucial in determining treatment options and prognosis. Clinicians conventionally rely on multiple Magnetic Resonance Imaging (MRI) sequences for accurate glioma assessment. Traditional deep learning methods typically utilize the individual MRI sequence with the annotations of Region of Interest (ROIs), incurring substantial manual effort while losing the complementary information provided by other sequences. This task is inherently a standard multi-modal problem. However, existing methods often adopt complex multi-stream networks for feature extraction and fusion, leading to high resource demands and potential feature redundancy. To address the above challenge, a discrepancy-aware self-distillation method is proposed for multi-modal glioma grading, which requires only a single-stream network to concurrently analyze multiple MRI sequences without auxiliary ROIs. The first Modality Discrepancy-aware Fusion (MDF) module considers the MRI imaging differentiation and widens the inter-modal contrasts, allowing the modality-specific features to be highlighted and the modality-invariant features to be diminished. Furthermore, the proposed Class Activation Self-Distillation (CASD) strategy leverages the generated Class Activation Maps (CAMs) as dark knowledge for the distillation process. This guides shallow layers to focus on the category discriminative features specifically within the lesion region. Extensive experiments were conducted on the BraTS2018 and BraTS2019 datasets to evaluate the effectiveness of our method. The results show that our method outperforms other relevant glioma grading and multi-modal fusion methods on both datasets.

Probabilistic U-Net model observer for the DDC method in CT scan protocol optimization.

Optimizing complex imaging procedures within Computed Tomography,
considering both dose and image quality, presents significant challenges amidst rapid
technological advancements and the adoption of Machine Learning (ML) methods.
A crucial metric in this context is the Difference-Detailed Curve, which relies on
human observer studies. However, these studies are labor-intensive and prone to
both inter- and intra-observer variability. To tackle these issues, a ML-based model
observer utilizing the U-Net architecture and a Bayesian methodology is proposed. In
order to train a model observer unaffected by the spatial arrangement of low-contrast
objects, the image preprocessing incorporates a Gaussian Process-based noise model.
Additionally, Gradient-weighted Class Activation Mapping is utilized to gain insights
into the model observer's decision-making process. By training on data from a
diverse group of observers, well-calibrated probabilistic predictions that quantify
observer variability are achieved. Leveraging the principles of Beta regression,
the Bayesian methodology is used to derive a model observer performance metric,
effectively gauging the model observer's strength in terms of an 'effective number of
observers'. Ultimately, this framework enables to predict the DDC distribution by
applying thresholds to the inferred probabilities.

Investigating the use of deep learning models for land cover classification from street‐level imagery

AbstractLand cover classification mapping is the process of assigning labels to different types of land surfaces based on overhead imagery. However, acquiring reference samples through fieldwork for ground truth can be costly and time‐intensive. Additionally, annotating high‐resolution satellite images poses challenges, as certain land cover types are difficult to discern solely from nadir images. To address these challenges, this study examined the feasibility of using street‐level imagery to support the collection of reference samples and identify land cover. We utilized 18,022 images captured in Japan, with 14 different land cover classes. Our approach involved using convolutional neural networks based on Inception‐v4 and DenseNet, as well as Transformer‐based Vision and Swin Transformers, both with and without pre‐trained weights and fine‐tuning techniques. Additionally, we explored explainability through Gradient‐Weighted Class Activation Mapping (Grad‐CAM). Our results indicate that using a Vision Transformer was the most effective method, achieving an overall accuracy of 86.12% and allowing for full explainability of land cover targets within an image. This paper proposes a promising solution for land cover classification from street‐level imagery, which can be used for semi‐automatic reference sample collection from geo‐tagged street‐level photos.

Classification of Chip-Level Defect Types in Wafer Bin Maps Using Only Wafer-Level Labels

Abstract Defective chips in wafer bin maps (WBMs) form different spatial patterns depending on the root causes of process failures. Therefore, the identification of defect patterns in WBMs can help practitioners identify the root causes. Previous studies have focused on wafer-level classification even though chip-level classification can provide additional information regarding defect locations and defect sizes. Chip-level classification is more challenging than wafer-level classification because existing chip-level classification methods require chip-level labels, which are laborious to collect. We propose a method for chip-level defect classification using only wafer-level labels based on weakly supervised semantic segmentation. We first train a classification network using wafer-level labels and extract class activation maps (CAMs), which are visualizations of the discriminative regions. We then generate chip-level pseudo-labels using the extracted CAMs and use these labels to train a segmentation network, which predicts chip-level defect types. Experimental results verify the effectiveness of the proposed method.

S-COPHY: A deep learning model for predicting the chemical class of compounds as cosmetics or pharmaceuticals based on single 3D molecular images

Non-animal-based in vitro and in silico approaches for the safety assessment of cosmetic ingredients, recently referred to as Next Generation Risk Assessment (NGRA)/New Approach Methodologies (NAMs), are evolving rapidly as approaches to provide a basis for the regulatory acceptance of new materials. However, predictive models should be applied only to chemicals within the chemical space defined by the dataset used in generating the model. Thus, only predictions for new molecules that are relatively similar to the modeling set can considered reliable with strong confidence. In this study, we developed the S-COPHY model, which employs deep learning to classify new compounds based on their structural similarity to a large collection of pharmaceutical and cosmetic compounds. S-COPHY shows high predictive accuracy both internally and externally, and in particular, there were only a few instances where pharmaceuticals were incorrectly predicted as cosmetics. The use of deep learning enabled the automatic generation of input data from SMILES (Simplified Molecular Input Line Entry System) information, resulting in more consistent model outcomes. Furthermore, GRAD-CAM (Gradient-weighted Class Activation Map) analysis provided insights into the specific structures that contribute to the model's predictions. The potentiality of S-COPHY to identify characteristic structures associated with pharmaceutical-like activity indicates its potential value in supporting safety assessments of cosmetic ingredients. Our results indicate that the S-COPHY model is a promising approach to support decision-making in large chemical spaces, thereby contributing to the safety evaluation of cosmetic ingredients. Expansion of the model to other categories, such as pesticides, could further extend its applicability.